Proactively searching the Web: Can we agree on what’s relevant?

We administered a survey to students from two populations in order to better understand how to best design the Watson information access assistant we are developing. Watson works by analyzing documents its users are manipulating in order to automatically query online information repositories and deliver information relevant to a user in the context of her current task. We found that users mostly agreed on the relevance of information retrieved by the software. In addition, contrary to our initial expectations, we found no connection between the user’s judgments of queries the system produces and the user’s evaluation of the results the system gathers as a result of executing those queries. These findings have direct implications for the design of the software and future evaluation methodology.